feat(evals): eval the tables and data guide in supabase.com/docs - #264

Closes DOCS-1304

Problem

database/tables is the most foundational database page in the docs and it is red: roughly 46 percent negative across its feedback votes, on the highest vote count of any task-driven database page. It teaches table creation competently — primary keys get bigint generated always as identity, foreign keys get a section, the private schema example is correct — and it never once says to protect a table. row level security appears one time in 573 lines, in the security_invoker note about views. No alter table ... enable row level security, no policy, no auth.uid().

Feedback intake carries the consequence as a sustained ask: FDBKIN-2454, FDBKIN-33286, FDBKIN-30897, FDBKIN-32570, FDBKIN-25441, FDBKIN-5065, FDBKIN-5041, FDBKIN-28984, plus four more closed as duplicates. Customers keep asking the platform to close this by default, which says the documentation does not close it.

Solution

• Adds build-docs-006-database-tables, suite: regression, services: [gotrue, kong, postgrest].
• The one claim: a page that teaches table creation and never mentions row level security leaves an agent creating world-readable tables.
• The prompt is a product request for a habit tracker plus the page url. A second clause gives it a starter list anyone can browse before signing up, which is what stops one approach satisfying every case: a solution that locks every table down fails the library check, and a solution that leaves everything open fails the private-data checks.
• The seed ships no migrations, because the schema is the subject. It carries the contract in local/src/queries.ts instead — the table and column names the app already builds against. Names avoid the ones a memorized reading of "habit tracker" reaches for, so an agent that skips the seed produces tables the app cannot query. routines.id has no fixed type on purpose; uuid and bigint generated always as identity are both correct, so the scorer resolves routine ids by title rather than inventing one.
• Eight checks: two read the catalog, four probe behavior over the Data API as a signed-out visitor and as the owner, one is the marker-row control the four probes are gated on, and one proves the page was read.
• Marker rows are written after the agent's code exists, scoped to the run, so a hardcoded array cannot clear a read control and no literal leaks between runs.

Two checks are deliberately not what you might expect.

anon holds no write grant is absent. Supabase's default privileges grant anon write on new public tables and the standard pattern leaves them in place while row level security gates the rows, so asserting the grant is gone would fail the canonical correct answer. a signed-out visitor cannot create a routine measures the outcome instead.

the signed-in owner reads their own routines and nobody else's is present so that enabling row level security and writing no policy cannot score as protection. A schema that refuses everyone is not protected, it is broken.

Verification

Five fixtures scored against the stack before any agent ran, with the verdicts written down first. Every one failed exactly what it was built to fail.

page-faithful is the finding. It is the page's own idioms and nothing else, and it hands every routine to a signed-out visitor to read and to write. The leaked titles in its notes are the scorer's own run-scoped markers, so that is a real leak rather than something the fixture hardcoded.

nothing-built is why the probes cannot pass vacuously. Without the control gate, an empty database satisfies "cannot read" and "cannot create" for free.

One prediction was wrong, and it was the check's fault

policies-to-public first scored 5/8, passing the anon-write check on a wide-open table. The probe sent only owner_id and title; cadence is not null in that schema, so Postgres rejected the insert as a not-null violation and the check counted any error as a refusal.

Fixed by making the probe send the whole contract row, so the only thing left that can refuse it is access control; by confirming the row's absence as the superuser rather than inferring it from the error; and by failing loudly with could not measure on any error outside 42501. The same guard now covers the read probe. The constraint is recorded in README.md so the insert does not get narrowed again.

A risk worth naming

This measures a docs gap that customers are asking the platform to close (FDBKIN-2454, FDBKIN-30897). If the default grants for anon on new public tables ever change, the central check saturates and the finding goes stale. That is the "check measures the platform, not the page" trap, better named here than discovered on a CLI bump.

Manual testing

1. Confirm the ports are free.

   docker ps --format '{{.Names}}\t{{.Ports}}' | grep -E '5432[0-9]->'
   

2. Confirm the eval is discovered and its metadata validates.

   pnpm eval:dry
   

   18 planned pairs, mode=local-stack, including the -no-skills variants.

3. Typecheck. Clean.

   pnpm typecheck
   

4. Format the eval's own paths, since Biome walking local artifacts fails on fixtures that are nobody's diff.

   npx biome check evals/build-docs-006-database-tables
   

5. Score the fixtures against the stack. Fixtures are git-excluded, so rebuild them from the table above.

   node --import tsx/esm ../../evals/build-docs-006-database-tables/solutions/score-stack.ts green page-faithful